Learning-to-Rank for Real-Time High-Precision Hashtag Recommendation for Streaming News

We address the problem of real-time recommendation of streaming Twitter hashtags to an incoming stream of news articles. The technical challenge can be framed as large scale topic classification where the set of topics (i.e., hashtags) is huge and highly dynamic. Our main applications come from digital journalism, e.g., for promoting original content to Twitter communities and for social indexing of news to enable better retrieval, story tracking and summarisation. In contrast to state-of-the-art methods that focus on modelling each individual hashtag as a topic, we propose a learning-to-rank approach for modelling hashtag relevance, and present methods to extract time-aware features from highly dynamic content. We present the data collection and processing pipeline, as well as our methodology for achieving low latency, high precision recommendations. Our empirical results show that our method outperforms the state-of-theart, delivering more than 80% precision. Our techniques are implemented in a real-time system, and are currently under user trial with a big news organisation.